View Synthesis by Appearance Flow

Given one or more images of an object (or a scene), is it possible to synthesize a new image of the same instance observed from an arbitrary viewpoint? In this paper, we attempt to tackle this problem, known as novel view synthesis, by re-formulating it as a pixel copying task that avoids the notorious difficulties of generating pixels from scratch. Our approach is built on the observation that the visual appearance of different views of the same instance is highly correlated. Such correlation could be explicitly learned by training a convolutional neural network (CNN) to predict appearance flows – 2-D coordinate vectors specifying which pixels in the input view could be used to reconstruct the target view. We show that for both objects and scenes, our approach is able to generate higher-quality synthesized views with crisp texture and boundaries than previous CNN-based techniques. Fig. 1. Given an input view of an object (left) or a scene(right), our goal is to synthesize novel views of the same instance corresponding to various camera transformations (Ti). Our approach based on learning appearance flows is able to generate higher-quality results than the previous method that directly outputs pixels in the target view [1].